feat(RFC): Adds altair.datasets
#3631
Draft
+3,955
−12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Allow quickly switching between version tags #3150 (comment)
To support [flights-200k.arrow](https://github.com/vega/vega-datasets/blob/f637f85f6a16f4b551b9e2eb669599cc21d77e69/data/flights-200k.arrow)
Not required for these requests, but may be helpful to avoid limits
As an example, for comparing against the most recent I've added the 5 most recent
- Basic mechanism for discovering new versions - Tries to minimise number of and total size of requests
Experimenting with querying the url cache w/ expressions
- `metadata_full.parquet` stores **all known** file metadata - `GitHub.refresh()` to maintain integrity in a safe manner - Roughly 3000 rows - Single release: **9kb** vs 46 releases: **21kb**
- Still undecided exactly how this functionality should work - Need to resolve `npm` tags != `gh` tags issue as well
Opened (narwhals-dev/narwhals#1897) Marking (#3631 (comment)) as resolved
- Shorter names `Read`, `Scan` - The single unique method is now `into_scan` - There was no real need to have concrete classes when they behave the same as parent
Resolves: ```py File ../altair/.venv/Lib/site-packages/pyarrow/csv.pyx:1258, in pyarrow._csv.read_csv() TypeError: Cannot convert dict to pyarrow._csv.ParseOptions ```
Also simplifies and removes outdated `Extension`-related tooling
- Reduced the scope a bit, now just un/supported - Added `pprint` option - Finished docs, including example pointing to use `url(...)`
dangotbanned
commented
Jan 31, 2025
dangotbanned
commented
Jan 31, 2025
Comment on lines
+294
to
+304
| # TODO: Open an issue in ``narwhals`` to try and get a public api for type conversion | ||
| def schema_pyarrow(self, name: _Dataset, /): | ||
| schema = self.schema(name) | ||
| if schema: | ||
| from narwhals._arrow.utils import narwhals_to_native_dtype | ||
| from narwhals.utils import Version | ||
|
|
||
| m = {k: narwhals_to_native_dtype(v, Version.V1) for k, v in schema.items()} | ||
| else: | ||
| m = {} | ||
| return nw.dependencies.get_pyarrow().schema(m) |
There was a problem hiding this comment.
TODO
- Open an issue in
narwhalsto try and get a public api for type conversion ([Enh]:nw.(DType|Schema)conversion API narwhals-dev/narwhals#1912) - feat: add
nw.Schema.to_*methods narwhals-dev/narwhals#1924
- Will be even more useful after merging vega/vega-datasets#663 - Thinking this is a fair tradeoff vs inlining the descriptions into `altair` - All the info is available and it is quicker than manually searching the headings in a browser
dangotbanned
commented
Feb 2, 2025
dangotbanned
added a commit
that referenced
this pull request
Feb 3, 2025
dangotbanned
added a commit
that referenced
this pull request
Feb 5, 2025
Contains a fix (narwhals-dev/narwhals#1934) for #3631 (comment)
Possible since narwhals-dev/narwhals#1930 @MarcoGorelli if you're interested what that PR did (besides fix warnings 😉)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related
Tracking
Waiting on the next
vega-datasetsrelease.Once there is a stable
datapackage.jsonavailable - there is quite a lot oftools/datasetsthat can be simplified/removed.3.0.0Release vega-datasets#654Discovered a bug that makes some handling of expressions a little less efficient.
Iterator[IntoExpr]narwhals-dev/narwhals#1897Upstreaming some
nw.Schemastuff tonarwhalsnw.(DType|Schema)conversion API narwhals-dev/narwhals#1912Description
Providing a minimal, but up-to-date source for https://github.com/vega/vega-datasets.
This PR takes a different approach to that of https://github.com/altair-viz/vega_datasets, notably:
vega-datasets/datapackage.jsonpandas"polars"backend, the slowest I've had on a cache-hit is 0.1s to loadExamples
These all come from the docstrings of:
LoaderLoader.from_backendLoader.__call__